The phrase "growing frustration" means that someone's feeling of annoyance or disappointment is getting worse over time.